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ABSTRACT: This commentary reflects on the contributions to learning analytics and theory by a 
paper that describes how multiple theoretical frameworks were woven together to inform the 
creation of a new, automated discourse analysis tool. The commentary highlights the 
contributions of the original paper, provides some alternative approaches, and touches on issues 
of sustainability and scalability of learning analytics innovations. 
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Kelly, Thompson, and Yeoman (2015, this issue) present a new, automated discourse analysis technique 
designed to assist with the orchestration of collaborative learning situations and informed by theories 
from the fields of learning analytics and CSCL. The automated discourse analysis technique they present 
is provided as an example of a theory-based development process. The authors are to be commended 
on their valuable contribution to the literature on learning analytics and theory. 


As the authors imply, there are advantages that accrue to the development of analytic tools from a 
strong theoretical base rather than the more common but less powerful atheoretical approach to such 
development. Off-the-shelf approaches, as they point out, are burdened with their own theoretical 
biases. The authors' approach of theory-led design, in which theory is used to establish first principles 
that then inform the development of tools, focuses on establishing function, behaviour, and structure, 
and then iterating through design improvements. This is an entirely reasonable and coherent approach. 
The authors identify two main frameworks that they use to inform their work: IRAC (Jones, Beer, & 
Clark, 2013) and the "general framework for learning analytics" proposed by Greller and Drachsler 
(2012). These frameworks are useful for helping us to understand the development of learning analytics 
tools. In particular, they help us to understand how learning analytics tools are developed. They do not, 
however, do as good a job at informing us about why learning analytics tools are developed. The 
question of "why" is framed in terms of theories of group cognition (Stahl, 2006) and orchestration 
(Dillenbourg & Jermann, 2010) that come from the field of CSCL. Overall, the authors have successfully 
woven together two broad theories about how learning analytics tools are developed with two broad 
theories that explain why particular learning analytics tools are developed. The resulting theoretical 
space is used to frame the development of their automated discourse analysis tool. 


An alternative approach to using theory-led, principle-based design that weaves together multiple 
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theoretical frameworks is to frame the work in terms of a single paradigm. My own involvement for 14 
years with the Institute for Knowledge Innovation and Technology provided deep enculturation in 
principle-based tool development. Starting with a series of deep principles about the process of 
knowledge building (Scardamalia, 2002), we developed software that represented technological 
supports for those principles. An off-the-shelf approach would have been considerably less expensive 
but would have introduced technological biases that run contrary to the unpinning theoretical 
principles. Over many years, the interplay between the technology (Knowledge Forum) and the theory 
of knowledge building has resulted in improvements to both. The knowledge building communities that 
have been engaged have also demonstrated impressive gains. 

It would have been nice to contemplate how the exploration of the tool and technique presented in the 
paper could have been used to inform improvement of the theories used to underpin the development 
of those tools. For example, are there potential improvements or enhancements to the IRAC framework 
as a result of the researchers' experience with the development of their tool? Flow does the use of their 
tool make us rethink group cognition? This feedback is a hallmark of design-based research and 
represents an extremely challenging framework from which to operate. Flowever, it also represents a 
clear route forward for the development and improvement of theory in the field of learning analytics, 
which is keenly needed. The field of learning analytics overall would be strengthened by having a suite 
of evolving theories that inform design and that are in turn tested and improved through empirical 
testing of predictions stemming from those theories. 

One question that emerges has to do with the sustainability of the development of theory- and 
principle-based tools and techniques. Are there better models for getting to scale with educational 
innovations? Those of us at the University of Michigan's new Digital Innovation Greenhouse think there 
are. Researchers are good at innovating but their creations can seldom be scaled up for widespread 
adoption. They are focused on creating and testing innovations but typically have little experience with 
developing software that can be supported as infrastructure. On the other end of the spectrum, 
Information Technology Services (ITS) organizations are very good at staging and supporting mature 
software systems. Their skills are ill-matched to the loose, rapid, duct-tape development methods of 
researchers. It is not possible to take code from a research group and hand it off to ITS for staging at 
scale. We have tried for several years; the mismatch is too large. This chasm between innovation and 
infrastructure is present in all kinds of technology transfer. Our team posited that higher education 
needs a greenhouse for propagation; an interim space that understands both why innovations arrive so 
fragile and how to make them stronger before they're taken "outdoors." In the world of entrepreneurial 
business, these spaces are often called "incubators." We have adopted an "incubator" model that seeks 
to take educational technology and learning analytics innovations and grow them to scale. Time will tell 
how successful we are. Innovators such as Kelly, Thompson, and Yeoman can help usher in a new era of 
learning analytics tools and techniques that incorporates not only powerful design frameworks but also 
concern themselves with the design, development, and deployment of robust and scalable tools and 
techniques. 
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